The quality of images formed by lenses is limited by the blur generated during the exposure. Blur most often occurs on out-of-focus objects or due to camera motion. While these kinds of blur can be prevented by adequate photography skills, there is a permanent intrinsic blur caused by the optics of image formation lens aberration and light diffraction.
Image deconvolution can reduce this intrinsic blur if the lens PSF is precisely known. The point spread function (PSF) can be measured directly using laser and precision collimator or pinhole image analysis. However, these approaches require sophisticated and expensive equipment. Modeling the PSF by means of camera lens prescription (see Y. Shih, B. Guenter, and N. Joshi, “Image enhancement using calibrated lens simulations”, European Conference on Computer Vision (ECCV), pages 42-56, 2012) or parameterized techniques (J. Simpkins and R. L. Stevenson, “Parameterized modeling of spatially varying optical blur”, Journal of Electronic Imaging, 23(1):013005-013005, 2014) is also possible. Unfortunately, these techniques are often applicable only for certain camera configurations and need fundamental adjustments for various configurations.
Hence, there is a requirement to measure the blur function by analyzing the captured images. Such a PSF estimation is an ill-posed problem that can be approached by blind and non-blind methods. This problem is even more challenging for mobile devices since they have a very small sensor area that typically creates a large amount of noise.
Blind PSF estimation is performed on a single observed image (S. Cho and S. Lee, “Fast motion deblurring”, ACM Transactions on Graphics (SIGGRAPH), 28(5):145, 2009; R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman. “Removing camera shake from a single photograph”, ACM Transactions on Graphics (SIGGRAPH), 25(3):787-794, 2006; A. Goldstein and R. Fattal, “Blur-kernel estimation from spectral irregularities”, European Conference on Computer Vision (ECCV), pages 622-635, 2012; N. Joshi, R. Szeliski, and D. Kriegman, “PSF estimation using sharp edge prediction”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-8, 2008; A. Levin, “Blind motion deblurring using image statistics”, Advances in Neural Information Processing Systems (NIPS), pages 841-848, 2006; T. Michaeli and M. Irani, “Blind deblurring using internal patch recurrence”, European Conference on Computer Vision (ECCV), pages 783-798, 2014; L. Sun, S. Cho, J. Wang, and J. Hays, “Edge-based blur kernel estimation using patch priors”, International Conference on Computational Camera (ICCP), 2013; L. Xu and J. Jia, “Two-phase kernel estimation for robust motion deblurring”, European Conference on Computer Vision (ECCV), pages 157-170, Springer, 2010; T. Yue, S. Cho, J. Wang, and Q. Dai, “Hybrid image deblurring by fusing edge and power spectrum information”, European Conference on Computer Vision (ECCV), pages 79-93, 2014) or a set of observed images (M. Delbracio, A. Almansa, J. M. Morel, and P. Muse. “Sub-pixel point spread function estimation from two photographs at different distances”, SIAM Journal on Imaging Sciences, 5(4):1234-1260, 2012; W. Li, J. Zhang, and Q. Dai. “Exploring aligned complementary image pair for blind motion deblurring”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 273-280, June 2011; L. Yuan, J. Sun, L. Quan, and H.-Y. Shum. “Image deblurring with blurred/noisy image pairs”, ACM Transactions on Graphics (SIGGRAPH), 26(3):1-10, 2007). The features of the latent sharp image are modeled, and then the model is employed in an optimization process to estimate a PSF.
Given the knowledge that the gradient of sharp images generally follows a heavy-tailed distribution (E. Simoncelli, “Statistical models for images: compression, restoration and synthesis”, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems amp; Computers, volume 1, pages 673-678 vol. 1, November 1997), Gaussian (Y.-L. You and M. Kaveh, “A regularization approach to joint blur identification and image restoration”, IEEE Transactions on Image Processing, 5(3):416-428, March 1996), Laplacian (T. Chan and C.-K. Wong, “Total variation blind deconvolution”, IEEE Transactions on Image Processing, 7(3):370-375, March 1998), and hyper-Laplacian (A. Levin, P. Sand, T. S. Cho, F. Durand, and W. T. Freeman, “Motion-invariant photography”, ACM Transactions on Graphics (SIGGRAPH), pages 71:1-71:9, 2008).
In addition to these general priors, local edges and a Gaussian prior on the PSF are used in edge-based PSF estimation techniques (see S. Cho and S. Lee, “Fast motion deblurring”, ACM Transactions on Graphics (SIGGRAPH), 28(5):145, 2009; T. S. Cho, S. Paris, B. K. Horn, and W. T. Freeman, “Blur kernel estimation using the radon transform”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 241-248, 2011; N. Joshi, R. Szeliski, and D. Kriegman, “PSF estimation using sharp edge prediction”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-8, 2008; L. Xu and J. Jia, “Two-phase kernel estimation for robust motion deblurring”, European Conference on Computer Vision (ECCV), pages 157-170, Springer, 2010).
In general, blind PSF estimation methods are suitable to measure the extrinsic camera blur function rather than the intrinsic one.
Non-blind PSF estimation techniques assume that given a known target and its captured image, the lens PSF can be accurately estimated.
Zandhuis et al. (J. Zandhuis, D. Pycock, S. Quigley, and P. Webb, “Sub-pixel non-parametric PSF estimation for image enhancement”, IEEE Proceedings—Vision, Image and Signal Processing, volume 144, pages 285-292, 1997) propose to use slanted edges in the calibration pattern. Several one-dimensional responses are required that are based on a symmetry assumption for the kernel. A checkerboard pattern is used as the calibration target by Trimeche (M. Trimeche, D. Paliy, M. Vehvilainen, and V. Katkovnic, “Multichannel image deblurring of raw color components”, SPIE Computational Imaging, pages 169-178, 2005), and the PSF is estimated by inverse filtering given the sharp checkerboard pattern and its photograph.
Joshi's non-blind PSF estimation (N. Joshi, R. Szeliski, and D. Kriegman, “PSF estimation using sharp edge prediction”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-8, 2008) relies on an arc-shaped checkerboard-like pattern. The PSF is estimated by introducing a penalty term on its gradient's norm.
In a similar scheme, Heide et al. (F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses”, ACM Transactions on Graphics (SIGGRAPH), 2013) estimate the PSF using the norm of PSF's gradient in the optimization process. They propose to use a white-noise pattern rather than regular checkerboard image or Joshi's arc-shaped pattern as the calibration target. This method also constrains the energy of the PSF by introducing a normalization prior to the PSF estimation function.
Kee et al. (E. Kee, S. Paris, S. Chen, and J. Wang, “Modeling and removing spatially-varying optical blur”, IEEE International Conference on Computational Photography (ICCP), pages 1-8, 2011) propose a test chart that consists of a checkerboard pattern with complement black and white circles in each block. The PSF estimation problem is solved using least squares minimization and thresholding out negative values generated in the result.
A random noise target is also used in Brauers's PSF estimation technique (J. Brauers, C. Seiler, and T. Aach, “Direct PSF estimation using a random noise target”, IS&T/SPIE Electronic Imaging, pages 75370B-75370B, 2010). They propose to apply inverse filtering to measure the PSF, and then threshold it as a naive regularization.
Delbracio et al. show (M. Delbracio, P. Musé, A. Almansa, and J.-M. Morel, “The non-parametric sub-pixel local point spread function estimation is a well posed problem”, International Journal of Computer Vision, 96:175-194, 2012) that a noise pattern with a Bernoulli distribution with an expectation of 0.5 is an ideal calibration pattern in terms of well-posedness of the PSF estimation functional. In other words, pseudo-inverse filtering without any regularization term would result in a near optimal PSF. The downside of the direct pseudo-inverse filtering is that it does not consider the non-negativity constraint of the PSF. Hence, the PSF can be wrongly measured in presence of even a little amount of noise in the captured image.
These techniques rely strongly on an accurate alignment (geometric and radiometric) between the calibration pattern and its observation.
There is a need for a method that will overcome at least one of the above-identified drawbacks.
Features of the invention will be apparent from review of the disclosure, drawings and description of the invention below.